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I am pleased to forward the report of the Defense Science 
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OFFICE OF THE SECRETARY OF DEFENSE 
WASHINGTON, D.C. a0301 -3140 


3 0 m 1994 

defense SCIENCE 
BOARD 

Dr. Paul Kaminski 
Chairman, Defense Science Board 

Office of the Undersecretary for Acquisition and Technology 
The Pentagon 
Washington, D.C. 


Dear Dr. Kaminski: 

Enclosed is the final report of the Defense Science Board Task Force on Acquiring Defense 
Software Commercially. We were tasked to determine the conditions under which procurement of 
defense spftware can use commercial practices and to define needed changes to permit such use. 
In the attaehed report, we make specific recommendations with regard to DoD process credibility, 
software program management,, required expertise of DOD personnel using modern software 
practices, use and integration of commercial off-the-shelf software, DoD software acquisition 
practices, use of software architectures by DoD as a management tool, and DoD’s investment in the 
software technology base. 

Based on our review, we concluded that issues associated with defense software are 
applicable across the wide spectrum of DoD software intensive systems. However, to reap 
maximum benefit from improvements related to the myriad aspects associated with software, the 
DoD requires a more coordinated approach to the oversight of its diverse software capabilities and 
programs. 

To provide such Department-wide oversight and to facilitate implementation of the other 
specific recommendations contained within this repOi t, the Task Force recommends that the 
Secretary of Defense assign to the Under Secretary of Defense (Acquisition and Technology) the 
responsibility for DoD-wide software technology, policy, practices and acquisition. This 
centralized approach will best serve the Department for all software intensive systems and 
programs. To support the implementation of this recommendation and to ensure that the key 
stakeholders are participants in the process, the Task Force further recommends the formation of 
an Executive Council consisting of the appropriate principals from OSD, the Services, and the 
Defense Agencies. 

We thank all of the members and government advisors of this Task Force for their 
dedicated efforts and significant contributions to this study. 
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Executive Summary 


The Defense Science Board Task Force on Acquiring Defense Software Commercially recognizes 
that DoD systems are becoming increasingly dependent on the use of software as the mechanism for 
implementing operational capabilities. To adapt to changing military and national .security situations, DoD 
is more dependent than ever on its ability to modify mission software rapidly, often in near real-time. 
However, softwsTe remains the schedule and cost driver for the development and maintenance of many 
important defense systems. 

I 

In its review' of current DoD and commercial .software acquisition practices, the Task Force found 
notable differences, as evidenced in Appendix C. There are, however, many similarities between the 
various categories of DoD and commercial software systems. Although there are indications that 
commercial development efforts have achieved better predictability and lower costs, the Task Force noted a 
significant lack of credible, quantitative data to substantiate this assessment. 

In general, the Task Force concluded that DoD's investment in software requires greater DoD-w'ide 
management control and oversight in the coming years if the Department is to exploit the use of 
commercial software acquisition practices fully, as well as rapid advances in software technology. The 
following is a summary of selected findings and recommendations toward that end. 

Process Credibility: Current DoD practice is not compatible with commercial business 
practices. DoD should work to make necessary changes to acquisition regulations such as: 

• Having program managers manage 3 of 3 (price/schedule/functionality) but only constrain 2 of 3 

• Defining successful performance on contracts as delivering a solution (with predictable price, 
schedule and functionality) not adherence to government processes, procedures and specifications 

• Not requiring c-level specifications for software projects developed in Ada 

• Establishing mechanisms to allow both current ability to perform as well as past performance as 
key factors in source selection 

• Encouraging offerors to demonstrate as much functionality as possible as part of bid without 
eliminating domain knowledgeable competition 

DoD Program Management: DoD program management approaches discourage the use of 
commercial practices. Program managers lack incentives to tailor procedures to fit individual program 
needs or to develop “corporate” solutions (e.g., employ common architecture or common software 
components). DoD should establish and implement overarching s iftware life cycle guidelines more 
conducive to the use of commercial practices and products, such as: 

• Defining software architectures to enable rapid changes and reuse 

• Facilitating early system engineering and iterative development 

• Participating in development of commercial and international standards 

• Allowing the fielding of software directly from test beds with user consent 

• Requiring program managers to stay with programs at least through beta testing to maintain 
continuity of understanding of original nuances in requirements 









DOD Personnel; There is currently a shortage of sufficiently qualified software personnel at all 
levels within the Department. DoD should establish a Department-wide software program management 
education and training initiative that includes: changing courses for PMs to reflect best commercial 
practices and other recommendations of this task force and providing for changes to reflect the dynamics 
of the software industry; rotating government and contractor personnel between PM and developer 
organizations to build understanding and trust; encouraging use of IPA's from industry; and integrating 
software-qualified personnel into senior DoD acquisition staff. 

Use and Integration of Commercial Off-the-Shelf (COTS) Software: DoD has not 
fully identified the pros and cons associated with the use of COTS software and, as a result, has not 
determined when and how best to use COTS software. To facilitate this process, DoD should require 
trade studies and analysis of the use of COTS software in DoD’s software acquisition process where 
appropriate. Further, DoD should establish “customer friendly," application-specific information 
technology “component stores” to enable program managers to assemble systems rather than develop them 
through use of reusable, piequalified components. DoD should also increase technology base funding for 
security audit tools for systems employing COTS software and should capitalize on innovative cost- 
effective techniques for acquiring and using COTS software products, such as the use of enterprise 
licenses. 

Software Architecture: Software architecture was emphr Ized by the Task Force as a means 
for achieving important ends. There is currently little emphasis on architecture in DoD software programs 
or regulations. As a result, DoD is not benefiting from architecture as a key tool for evolutionary 
development and for early (and frequent) involvement of users with functional capability and facilitating 
reuse, requirements changes with minimum cost and schedule, and “product line” management. DoD 
should require vendors to propose, manage and control the architecture and should establish an early 
architecture deliverable in all developments. 

Software Technology Base: The current DoD software technology base investment does not 
adequately take advantage of commercial R&D. Further, software technology transfer (both internal and 
e.xternal) is not receiving adequate emphasis within DoD. DoD should provide for the evolution of the 
DoD Software Technology Strategy to align with emerging commercial technology and practices. 

Overarching Recommendations: To facilitate implementation of the many recommendations 
contained in this report, the Task Force concluded that DoD's investments in software require greater 
management control and DoD-wide oversight. To this end, the Task Force recoirmiends that the Secretary 
of Defense (SECDEF) assign the Under Secretary of Defense (Acquisition and Technology) (USD (A&T)) 
the responsibility for DoD-wide software technology, policy, practices and acquisition. In can 7 ing out 
this responsibility, the Task Force recommends that the USD (A&T) consider forming an executive 
council with the Director, Defense Research and Engineering (DDR&E), the Assistant Secretary of 
Defense (Command, Control, Communications and Intelligence) (ASD (C3I)) and appropriate 
representatives from the Services and Defense Agencies as members. Further, USD (A&T) should 
provide for a supporting “process action team” to assist in implementation of the recommendations of this 
Task Force study. 




Defense Science Board Task Force 
on Acquiring Defense Software Commercially 


1.0 TASK FORCE OVERVIEW 
1.1 TERMS OF REFERENCE 



Terms of Reference 



Objectives: 

• Detennine: 

- Conditions Under which Procurement of Defense Software Can Use Commercial Practices 

- Changes Required to Permit Such Use 

• Develop Strategy that Incorporates Such Practices 

- Not Constrained by Existing DoD Standa'ds 

- Viewed as Coexisting Alternative Strategy 

- Includes DoD Use of Commercial Software Products 

• Compare Proposed Strategy with Current DoD Strategy, Indicating 
Circumstances Where Each is Most Beneficial 

Scope: 

• All Software Intensive Systems 

• AH Stages of the Software Life-Cycle 




Appendix A provides the Terms of Reference by which the Task Force was established. The 
objectives and scope of this Task Force are cutiined above. At the first meeting of the Task Force, the 
sponsor of the study (the Director, Defense Research and Engineering) requested that the Task Force 
provide a strategy that: was sensible, pragmatic, and unconstrained by current DoD acquisition practices; 
was based on evduation of mechanisms for integrating defense software efforts with commercial software 
efforts, did not require legislative relief; and addressed the full spectrum of DoD software applications. 
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1.2 CAVEATS 


Caveats 


• Relied on Inputs from DoD and Industry Experts 

• Provided No Detailed, Quantitative Assessment or 
Evaluation of Individual Topics 




• Did Not Address: 

- Success or Failure of Ada 

- Recommended Software Development Approach for Specific 
Programs 

- Specific COTS Products for DoD to Exploit 

- Trade-Offs Between Hardware and Software 

_ 



The Task Force relied heavily on inputs from a variety of DoD and industry experts regarding a 
wide range of topics related to defense software technology, policy, practices and acquisition. Appendix 
B lists the many briefings that were provided. Although the Task Force chose not to provide detailed, 
quantitative assessments or evaluation of individual topics, it used the information associated with these 
topics in the formulation of findings and recommendations. The areas not addressed by the Task Force, 
while important in and of themselves, were determined not to be directly gennane to the development of an 
overall defense software strategy. 
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In its efforts to assess the appropriateness of DoD use of commercial practices, the Task Force 
addressed software management, contracting, and technical issues. The above viewgraph lists the primary 
topics considered within each of these categories. 
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1.4 MEMBERSHIP 
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Task Force members represented a valued cross section of software expertise within both the DoD 
and commercial sectors. The government advisors represented senior executives (including the three 
Service Software Executive Officials) from the major software management organizations within the 
Department. 
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2.0 CURRENT POD AND COMMERCIAL SOFTWARE ACOUTSITION PRACTICES 


2.1 BACKGROUND 



Background: Previous Studies 




DSB Summer Study on Technology Base (1981) 

Joint Service Task Force on Software Problems (1982) 

AF SAB High Cost and Risk of Mission Critical Software (1983) 

CODSIA Report on DoD Management of Mission-Critical Computer Resources (1984) 
DSB Task Force on Military Software (1987) 

Ada Board Response to DSB Task Force (1988) 

Summer Report on Defense-Wide Audit of Support for Tactical Software (1988) 
Workshop on Executive Software Issues (1988) 

Workshop on Military Software (1988) 

Army Materiel Command Study (1989) 

Software Technology Development and Deployment Plan for DoD Technology Base 
(1989) 

AF SAB Adapting Software Development Policies to Modem Technology (1989) 

Draft DoD Software Master Plan (1990) 

Draft DoD Software Technology Strategy (1991) 

AF SAB Study on Information Architecture (1993) 

Study on Military Standards Impacts on the Acquisition Process (1993) 

Draft Software Action Plan Working Group Report (1993) 

Evolutionary Acquisition Study, AFCEA (June 1993) 


I 



As a point of departure, the Task Force noted a number of previous studies addressing issues 
related to the defense software technology, policy, practices and acquisition. Despite the increased 
emphasis given to software issues by the DoD (as evidenced by the above list), the majority of the 
recommendations resulting from these studies have not been implemented. 
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Background: Impediments to Change 


• DoD Software Management 

No Single Operative Mechanism Exists to Implement Change 
Three Separate Domains: MIS, C3I, Embedded 
Two Separate Acquisition Management Structures 
Bureaucratic "Turf" Inhibits Real Reform 

• Acquisition Process 

5000 and 8000-Series Developments Typically Employ "Waterfall" 

Approach; Not Incremental/Spiral Approach 

• Culture 

Systems Are Stove Pipe — My System, My Program (PM is King) 

DoD Acquisition Training Reinforces the Wrong Approaches 

• Procurement 

The Contracting Process Inhibits Creativity and Investment by 

Contractors; Limits Options 

Interpretation of Competition in Contracting Act 

V_ y 

To ensure that its recommendations could be readily implemented by the DoD, the Task Force 
identified the primary reasons why recommendations from previous studies had not been acted upon. The 
Task Force then foimulated its recommendations to appropriately address these impediments. 
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Why Is This Study Important? 




• DoD Military Capability Increasingly Software Dependent 

• Software One of Few DoD Budget Areas That is Projected to Grow 

- Software Has Become the Overall Defense System Schedule and Cost Driver 

» Declining DoD Budget — Affordability a Major Concern 
» Escalating Costs of Software Development and Life Cycle 

- Significant Level of DoD Resources To Be Hpent on Information Technology: 

>. FY92-FY94: ~$10 BillionA'ear* 

» ElA Forecast for FY9S-FY98: ~$10 BillionA'ear 
» In-House vs. Contracted Out: 30% In; 70% Out 

- Will Require Greater Management Control 

• Rich Commercial Base to Tap; Many Opportunities 

- Custom Software - Acquisition Methodologies 

- Off-the-shelf Products - Approach to Requirements Determination 

• Functionality and Flexibility More Embedded in Software than Hardware 

• Study is Timely Because DoD Leadership is Focused on Acquisition 
Reform 

- Something May Actually Get Done 


Given its increasing reliance upon software as the mechanism for implementing system 
capabilities, coupled with the rising costs associated with software development and maintenance, DoD 
must take action now to address the issues associated with software acquisition. The commercial sector 
provides numerous opportunities upon which the DoD could readily capitalize. The time is ripe for 
assertive DoD action, particularly since the current leadership is so strongly focused on acquisition reform. 
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The Software Domain 
is Very Large 


CUSTOM 
COMMERCIAL 
REAL TIME SYSTEMS 


LARGE DOD EMBEDDED REAL 
TIME SYSTEMS 


COMPLEXITY 



COMMERCIAL 

ENGINEERING 

SYSTEMS 


SMALL DOD 
EMBEDDED REAL 
TIME SYSTEMS 


DOD 

ENGINEERING 

SYSTEMS 


CUSTOM DOD 
BUSINESS SYSTEMS 


CUSTOM COMMERCIAL 
BUSINESS SYSTEMS 


COMPLEX 

COMMERCIAL 

PRODUCTS 


=fT 


r - 


COTS 

RELEASES 

u 


SMALL 

COMMERCIAL 
COTS PRODUCTS 


COST TO DEVELOP 




The software domain reviewed by the Task Force encompasses a wide variety of DoD systems, 
ranging from software tools to large embedded real-time, systems. The associated cost, complexity, and 
time required to develop these systems varj' widely, both for commercial and military applications. 
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Background: 

DoD Software Acquisition Management 



• Two Different Sets of Software Policies/Rules Managed by 
Two Different Organizations 

- USD (A&T) for Software Embedded in Weapons/Systems 

- ASD (C3I) for C3I Software and MIS Applications 


• Majority of Software Issues are Applicable to AH DoD 
Systems 

• Growing Similarity Between DoD and Commercial Software 
Developments Across the Various Types of DoD Software 

- Modem Software Tending to Blur Distinction Between DoD and 
Commercial Applications 





Central DoD Leadership Needed More than Ever 

- Major Revisions Needed in DoD Software Policies/Rules and 
Management Across All DoD Applications in Order to Meet SECDEF's 
Acquisition Reform Goals 

- Interpretation and Implementation of DoD-Wide Policies/Rules by 
Management is a Central Issue 



Today, the management of DoD software acquisitions is quite complex. There arc two sets of 
software policies/iules managed by two different organizations 

• USD (A&T) for software embedded in weapons/systems 

• ASD (C3I) for C3I software and MIS applications 


There currently exists within the DoD a dichotomous organizational structure for the management 
of DoD softv.'are intensive systems. The Under Secretary of Defense (Acquisition and Technology) is 
responsible for weapons systems; the Assistant Secretary of Defense (Command, Control, 
Communications, and Intelligence (C3I)) is responsible for C3I systems and Management Information 
Systems (MIS). However, as technology has evolved over the years, the issues associated with the use of 

owi.fcV^aiw Oa* C/a aiiwov ojotviiic iitfVv crwvviiiiv vddWilUaiJy UiC 1 i:>£>uaiiuc Ui Ul&IClCiU ^UltWalC” 

related policies and regulations that are oriented according to the type of system is no longer appropriate. 
If the DoD is to adequately address this fundamental issue, central focused management is required. 
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2.2 COMPARISON OF DEFENSE AND COMMERCIAL SOFTWARE 



Comparison of 

Defense and Commercial Software Needs 



• Defense and Commercial Software Applications Needs Are Merging 



Breadth of 
Application 

WKmSSsSMKM 


Sensitivity to 
Errors 


Unique 

400,000 

CpmpleK; 

Inflexible 

High 

- DoD Real Time Software 

- CoounrciAl Luatom bolUvare iniegrated 
Into Reat'Time OneMtiona 

Unique 

400,000 


muiH 


Moving Toward 
Wide 

200,000 

-1 

Complex; Flexible 

Moderate 

- C3I 

« commercial Cuatom software integrated 
Into Buiincat Operations 

Moving 1 oward 
Wide 

200,000 

IBsBEHBl 

Moderate 

MIS Systems 

« DoD Automated Information Systems 

Very Wide 

200,000 

Moving Toward 
Commercial 

Low 

x Commercial Automated Information 
Systems 

Very Wide 

200,000 

Exploits Object 
Oriented 
TechnoloKv 

Low 


VdtyWide 

50,000 

Tied to Wcapen 
System Cycle 

Low to High 
(Depending on Use) 

- DoD Component Stores 

> Commtrcial Shrink Wrapped 

Very Wide 

50,000 

Tied to 

Commercial CVeU 

U'w 


DoD and commercial applications for software are different in some ways but very similar in 
others. The above table highlights these similarities and differences for real-time applications, command 
and control applications, MIS systems and reusable components and products. This apparent disparity in 
the classification of software between DoD and commercial vendors increases the complexity of DoD's 
management task. Companies (people, methods, and organizations) are usually specialized to one or more 
commercial type systems, not to DoD type systems. Defense and commercial software applications needs 
are merging, providing the potential that DoD can exploit commercial capabilities more effectively over 
time. Major revisions are needed in DoD software management across all DoD applications in order for 
DoD to capitalize on the evolving commercial base. 
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Process Comparison Summaiy 
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As is evident from the process compai ison summary above, there are not only differences betv/een 
commercial and defense applications, but ^so in the process used by each to develop, acquire and support 
complex software systems. Defense and other federal acquisition regulations and detailed specifications 
require a much more complex set of dellvera..rles within the context of DoD’s contracting process. 
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Comparison of DoD and Commercial 
Software Acquisition Practices 


a Requirements Definition 

- Commercial 

- More flexible and open between users and supplier 

- Based on strategic plan And usually cost/schedule driven 

- More willing to adjust requirements based on availability of off-the-shelf 
products 

- Evolves capability 

« Vendor Selection 

- Commercial 

- Much more flexible; no requirement for fairness or tc maintain the public trust 

- Encourages vendors to offer best solution, not meet 100% of requiremertis 

- Accommodate teaming and long-term relationships 

• Development Process 

- Commercial 

- Mote flexible; product improvements anticipated 

- Team approach with bias toward reuse and tailoring of existing systems 

- Multi-year acquisitions rmt re-justif.led each year. 


I'n esscncB, the Task Force found major differences between DoD and commercial software 
acquisition practices, as outlined above and on the next page. 
















Comparison ofDoD and Commercial 
Software Acquisition Practices (Cont) 



9 Business Practices 

- Commercial practice more flexible with greater incentives 


• Integration Testing and Delivery 

- Commercial provides integration and functional testing according to need 

- DoD uses separate test agency with added time and complexity 

- Absence of beta testing within DOD increases costs 


o Rights in Data 

- Commercial more flexible, especially regarding resales 


• Maintenance 

- Commercial: Maintenance considered and integrated with development 

- DoD: Maintenance not major factor in development process 




Api^endix C piovides a more complete compaiison of DoD and commercial software acquisition 
practices as developed by this Task Force. 


I 
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Assessment of Current DoD 
Software Acquisition Approach 



• Strengths 

- Highly Structured Process Tied to Individual System 
Developments 

- Tightly Defined Requirements 

- Produces High Quality Product for Mission Critical Systems 
That Demand Extremely Low Failure Rates (e.g.. Flight 
Controls for Man-Rated Platforms) 




The strengths of the current DoD approach to software development, acquisition and operation are 
surrunarized above. The Task Force found that the highly structured DoD process has, in fact, provided a 
high quality software product, in most cases. 
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Assessment of Current DoD 
Software Acquisition Approach 


Weaknesses 

- High Life Cycle Cost in Time and Dollars 

- Software Development Cycle Tied to Weapon System Developments 

» Incredibly Long, Typically 13 to 15 Years from Concept to Fielding 

- Encourages Excessive Acquisition Agent Involvement in Design 
Detail and Process 

- Based on Mistrust vs. Mutual Trust 

» Excessive Documentation 
» Excessive Fomtal Review 
» Excessive Testing of Non-Critical Systems 

» Poor Communication Between Vendor, Acquisition Agent and User 

- No Requirements Addressing Cost and Schedule 

- Traditional Approach Used: Design it All and Then Build it 

- Little or No Requirements Relaxation for High Cost Items 

- Inadequate Beta Testing in Early Phase 

- Little Focus on Designing in Reusability 


However, there are a number of weaknesses associated with the current defense approach, as 
surnmari 2 :ed above. Many of these weaknesses derive from the need for a fair and open procurement 
process and the necessity to prove that public dollars are wisely spent. 
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Assessment of Current Commercial 
Software Acquisition Approach 

Strengths 

- Open Architecture Compatible with Usage of COTS Software 

- Much Less Formal Competitive Procurement Process 

» Includes Prior Performance as Major Selection Criterion 

- Trend Toward Joint Assumption of Risks Between Buyers and Suppliers 
Across Development, Operation, Maintenance and Modernization 

- Process is More Flexible 

- Shorter Cycle (Product Release) Times 

- Tailorable Level of Documentation and Oversight 

- Emphasis on Reuse and Tailoring Requirements to Existing Products 

- Beta Testing Widely Used 

Weaknesses 

- Less Responsive to Continual Changes in Requirements 

- Less Assurance that Software Will Function Properly Under All Situations 

- Potentially Locked into One Vendor's Proprietary Application 


J 


The strengths and weaknesses of the current commercial approach to software development, 
acquisition and operation are summarized above. 
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Principal Reasons DoD Software 
Programs Get Into Trouble 


• Poor Requirements Definition 

- Lack of User Involvemet\t in Development Process 

~ Inability of Users to Foresee Benefits of Automation Without Incremental Capability 

• Inadequate Software Process Management and Control by Contractor 

• Lack of Integrated Product Teams 

- Failure to Establish "Team" With Vendors and Users 

- Little Participation of Functional Area Experts 

• Ineffective Subcontractor Management 

• Lack of Consistent Attention to Software Process 

• Too Little Attention to Software Architecture 

• Poorly Defined and Inadequately Controlled Interfaces Between Computer 
Hardware, Communications and Software 

• Assumption That Software Upgrades Can "Fix" Hardware Deficiencies 
(Without Assessment of Cost and Schedule Risks) 

• Focus on Innovation Rather than Cost and Risk 

• Limited or No Tailoring of Military Specifications Based on Continuing 
Cost-Benefit Evaluations 


V. 


Over the last two decades, a number of DoD software development efforts have gotten into trouble, 
particularly in terms of actual costs and schedule vs. expected or predicted costs and schedule. The Task 
Force heard briefings from a number of DoD program managers where this was the case. Based on the 
specific programs discussed and on other inputs, the Task Force prepared a listing of the principal reasons 
DoD software programs have gotten into trouble as shown above, Certain of these reasons ai'e a reflection 
of DoD practice. Others are tied to the software engineering level available at the time such programs were 
initialed. The Task Force believes that today’s software technology and practices can directly address 
many of the root causes for such past problems. 









r 


Resource Allocation: 
Where Does the Effort Go? 


Soufce: Magitavox and Capers Jones 





Military 

Commercial 

• Engineering 

30% 

50% 

• Evaluation 

20% 

20% 

• Management 

15% 

10% 

• Meeting Support 

15% 

5% 

• Documentation 

15% 

7% 

• Customer/User Support 

5% 

8% 


There are some indications that commercial development efforts have achieved better predictability 
and lower costs than DoD counterparts. The Task Force solicited quantitative data on this issue from both 
government and commercial software experts. The figure above summarizes one type of indicator of the 
difference between commercial and government projects (in terms of the percentage of the effort expended 
for different aspects of a typical development). 
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^ Comparison of 

Commercial and Government Projects 


— 

Application Type 

— 

Average Size (SLOC- 
New and Modified) 

■nSSBlHl 


Commercial 

v'a ^ vN '' ' 

1491 

26,000 

12.52 

t V.'. i ^ ^)vky V -ii 

Government 

75 

26,000 

15.1 


Percent Difference 



+ 17.09% 

+ 44.24% 

Commercial 


26,000 


65.7 

Government 

29 

26,000 

20.9 

117.3 

Percjnl Difference 



+ 16.75% 

+ 43.99% 

Commercial 

295 

21^000 



Government 

21 

212,000 

256 

242.9 

Percent Difference 


_ 

+14.06% 

+ 38.16% 






Commercial 

56 

442,000 

45 

1736 

Government* 

7 

442,000 

52 

2740 

Percent Difference 



+ 13.46% 

+ 36.64% 

Summary of 

Averape Percent Difference 



+ 15.34% 

+ 40.76% 


* Small sample may be statistically invalid. 
Source: QSM, Inc. 


This figures summarizes other quantitative indicators of the difference between commercial and 
government projects (in terms of the typical size of the code, time to develop the application and overall 
level of effort). 

It should be noted that the Task Force was unable to find reliable, quantitative data supporting the 
notion that commercial practices are more cost-effective than DoD piactices. This lack of reliable 
indicators was a major concern of the Task Force. 
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5.0 MA lOR FINDINGS AND RECOMMENDATIONS 
3.1 PROCESS CREDIBILITY 



Process Credibility 



Findings 

• Attempts to achie\ e absolute frimess in competition in contracting 
(fair and reasonable pricing) have led to a lack of trust between 
government and individuals/contractors 

- Current DoD practice: 

» Is not compatible with commercial business practices 
» Is focused on contractor management/audit activities and costs 
» Hinders flexibility (managers take no pe.-onal risk) 

» Is not well suited to procuring complex, knowledge-intense, “first of a kind" systems 
» Is too costly 

>' Does not prevent malfeasance or incox..petence 
» I.eads to adversarial relationships 

» Reduces reuse and contractor incentive to invest because of stringent data rights 
interpretation 

- Contractors must make profit on single contract; no long term relationship (DoD 
business practices do not view profit as a legitimate cost of doing business) 

- No individual fully understands or owns total process 

\_ J 


The Task Force spent considerable effort on how the requirement for DoD to ensure public trust in 
its acquisition process influences DoD’s ability to employ “commercial best practices.” The Task Force 
found that attempts to achieve absolute fairness in competition in contracting have, in fact, led to a lack of 
trust between the government and individuals/contractors. DoD acquisition processes are focused f i 
contractor audit activities to an extent that hinders the flexibility of Program Managers and contractO: >. 
DoD PMs are not incentivized to assume any personal risk associated with allowing for flexibility 
comparable to commercial practice. Criminal sanctions are a significant disincentive for such efforts. 

DoD’s acquisition system still does not prevent malfeasance or incompetence and leads to 
adversarial relationships rather than partnerships which are the nonn within commercial industry software 
developments. One problem highlighted by the Task Force was that the current DoD system does not 
allow one individual or manager to contr il the total process, even for a specific project. This lack of 
control leads to a diffusion of accountability and hinders DoD’s ability to oversee complex “first of a 
kind” software developments. 


2 ) 




Process Credibility 



Finding s 

• Requirements and Source Selection Inflexibility 

- Vendors attempt to meet every requirement 

- Ease of protest 

- Requirements focus vendor on particular solution 

- Requirements have became alternative to professional judgment 

- Departures from requirements have caused protests and bad publicity 

• Price/Schedule/Functionality 

- In commercial sector, managers constrain 2 out of 3 (e.g., cost and schedule) 

- In DoD, managers constrain 3 out of 3 

• Constrained Communication During Solicitation 

- Questions are provided to competitors <can give away proprietary concepts) 

- Questions are often misinterpreted and answered incorrectly 

- Guarded way of asking questions limits substantive feedback 

• Complicated Regulations 

- Restrict variety of proposals 

- Restrict competition and limit government options 

- Consume time and resources 

- Drive government employees to follow conservative procedures as safest path (inhibits "best value") 

• Govemment'Unique Accounting Procedures and Audits 

- Audit requirements limit available vendors 

- Add substantial cost 

- Create need for separate accounting systems 

- Drive vendor to lowest labor cost solution rather than best value solution 

- Lead to rejection of good solutions based on value pricing vs cost-based pricing 


The viewgraph above lists important hindrances that the Task Force sees with regard to the 
adoption of commercial software practices. Many of the Task Force recommendations address these 
hintirances. 
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Process Credibility 



Recommendations 


• Make necessary changes to acquisition regulations 


- Have Program Managers Manage 3 of 3 (Price/Schedule/Functionality) But Only 
Constrain 2 of 3 

- Define Successful Performance on Contracts as Delivering Solution (with Predictable 
Price, Schedule and Functionality) Not Adherence to Government Processes, 
Procedures and Specifications 

■■ Do Not Require C-Level Specifications for Software Projects Developed in Ada 

- Prior to RFP, Government Should Perform Independent Market Analyses of Off-the- 
Shelf and Contractor Products to Assure "Best Value" Solution 


Establish Mechanisms to Allow Both Current Ability to Perform as Well as Past 
Performance as Key Factors in Source Selection 

u Require Source Selection EvaluaEon of Development Contractors Through a Formal Software 
Process Capability Evaluation 

Encourage Offerors to Demonstrate as Much Functionality as Possible as Fart of Bid 
Without Eliminating Domain Knowledgeable Competition 
» Executable Architecture as a Minimum 
» Weight Heavily in Selection 


The Task Force makes the above recommendations with regard to process credibility. 







3.2 DOD PROGRAM MANAGEMENT 



DoD Program Management 



Findings 

• Program management does not encourage "80% solution for 20% cost" 

• Users often not significantly involved in process 

• Little emphasis on life-cycle issues (including maintenance and support) 

• Existing policies, methodologies, and procedures, and their implementation are 
inadequate 

- Little evidence that policies are influenced by actual experience and vice versa 

- Little effort to measure effectiveness and costs of policy directives 

• Some indication that DoD is migrating toward development and use of 
standards-based architectures 



Program Managers lack incentives to allow tailoring of procedures to fit 
individual program needs or to develop "corporate" solution (e.g., employ 
common architecture or common software components) 



The Task Force found that the DoD program management system itself discourages the use of 
commercial best practices. These findings are not unique to software, but are particularly important for 
software given the significant cost savings associated with software reuse. There are few incentives for 
Program Managers (PMs) to develop or even employ corporate solutions (common architectures/software 
components), particularly if they are more expensive to acquire. Rather, each PM will tend to optimize 
his/her development for its own purpose. Further, it is difficult for users to be significantly involved until 
late in a software development process, unless some sort of prototype can be constructed. DoD PMs place 
little emphasis on life-cycle issues (such as softwaie maintenance and support). Existing DoD-wide 
software policies, methodologies, and procedures, and their implementation by PMs are inadequate. 
There is little evidence that policies are influenced by actual experience and vice versa and there is little 
effort to measure the effectiveness and costs of policy directives. 
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DoD Progtam Management (cont.) 


Recommendations 

• Establish Overarching Software Life Cycle Guidelines 

- Tools/Methods 

» Define Soflwere Arehilecturee to Eneble Ripid Oungee end Reuee 

» To Achieve the Bencfite of Using SLindards-Baeed AKhitectuice, DoD Mint Manege Programa Using: 

• E*riv ty*!*** cnBinMrinf 

• litMiivt dcvciapiMnl 

• Proactive participation In development of tHeee eUndatda 

>* Promote DtvdtopmonlAJH of Community-Wide Metrlci And Models (e.g.e SEI's Cepability Maturity Model) 

- Acquisition 

» Revise the Milestones for Softwsre-lntensive Development 

• Addreee the need fore Mflwerv-^flr*! phlleeophy 

• Provide for a layered eoOwaro/hatdtwaM ataitdarda haMd aicMterture 

• Ac^Hlalttein ervd life cycle pleimlng ahauld ttew aeparele herdvrare and aoflwaie fielding baaed on the bvelntea aenee of the apedfU prelect 

» Requiro Early Interaction Between User, Acquiaition Agent, and Developer; Identify and Get Early User Involvement 
» Apply Evolutionary Development with Rapid Deploynwnt of Initial Functional Capability 
u Encourage Competition of Tecluikal Approach vs. Cost 

» Provide Incentives and Guidelines to Encourage Software Reuse (Architecture-Based Reuse) 

M Reduce DcKumenUlion and Review Requirements for *Matuvc'' Companies (i.c., Companies Dotennlned to Be *^alure* 
Through Evaluation Mechaniema) 

» Tailor operational testing to develop DoD 'Beta TesP* philosophy 

• Altew fUldlna of Mftwert dlncl from VmI bode with MMroociMnl 
Have Program Manager Stay with Progvama at Leaat Throu^ Beta Testing to Maintain 
Original Nuances in Requirements 






The Task Force makes the recommendation that DoD establish overarching software lifecycle 
guidelines directed at facilitating program manager employment of commercial practices and software and 
that a DoD-wide effort be made to oversee implementation of these guidelines. The tools, methods and 
acquisition approaches recommended by the Task Force are listed above. 



3.3 DOD PERSONNEL 



DoD Personnel 


u» 



I 

I 


Finding s 

• A shortage of sufficiently qualified software personnel 
currently exists at all levels within the DoD 

- Expertise for software acquisitions, software evaluations, and software 
maintenance/support 

- Expertise to represent DoD (customer) interests with commercial sector 

- Expertise in domain software design and applications | 

- Expertise in software technology to develop policies, standards, and | 

guidelines 

- Expertise in software program management 

V_ J 


Based on the inputs provided, the Task Force found that a shortage of sufficiently qualified 
software personnel exists at all levels within the DoD. Without personnel who aie highly qualified in 
modem software practices, DoD will not be as capable of effectively exploiting complex software within 
its systems. This personnel shortage has been a major contributor to the problems that have arisen in past 
DoD software development programs. 







DoD Personnel (Cont.) 


Establish DoD-wide software program management education and training 

initiative 

- Change D5MC and IRMC courses for PMs to reflect best commercial practices and other 
recommendations of this Task Force and Provide for changes to reflect the dynamics of the 
software industry 

- Develop and provide interactive training tools for senior managers to perfect software 
management skills 

- Rotate government and contractor personnel between PM and developer organization to build 
understanding and trust; encourage use at IPA's from industry 

- Incorporate software management principles in senior management education and seminars 
(including senior services colleges) 

- Provide mechanisms for keeping software expertise current in the workplace 

Develop Acquisition Managers with software program management expertise 

- Integrate software-qualified personnel into senior acquisition staff 

Establish Norms for the Number of Software Experts in Program Offices 

Upgrade Educational Requirement for Personnel Assigned to Acquisition, 

Management, Development and Oversight of Software Intensive Programs 

Develop Expertise in Analysis of Domain Software Design 

- Promote Software Reuse in the Design 


The Task Force makes the above recommendations with regard to DoD software expertise of its 
personnel. The Task Force strongly recommends an emphasis on increasing the capability of its personnel 
in modem software practices and techniques. 




3.4 USE AND INTEGRATION OF COMMERCIAL OFF-THE-SHELF SOFTWARE 



Use and Integration of 
Commercial Off-the-Shelf Software 



Finding s: 

• DoD Does Not Normally Perform COTS Market Analyses Incident to 
Requirements Definition 

• DoD is Beginning to Exploit Evolutionary Development Approaches 

• Tools/Methodologies Are Evolving to Facilitate Use of COTS 

- Computer Aided Software Engineering (CASE) 

- Object oriented methods 

- Reuse repositories 

- Integrated product teams 

- Integrated risk management 

_ ) 


DoD does not routinely perform COTS software market analyses during the requirements 
definition phase of an acquisition program. Nor does it employ prototypes or simulations of new 
capability in a way that could influence the requirement process. Rather, requirements are evolved in a 
manner that is disconnected from the availability of commercial applications and are not typically 
influenced by such available capability. This situation is true for both hardware and software. Today’s 
technology can facilitate early interaction between users and available capability, particularly in software. 
Further, DoD is beginning to exploit evolutionary software development approaches that could provide for 
the inclusion of existing commercial software functionality into early prototypes and allow users to test 

<iUcH Canshilitie^i Viav^ iieo /^OT'C 

ivaWwWAaA iV/Xifav# •44V'VC* VrO AAMVV' VVv/AVWVJ 4I4UL A IkV'l 4 4 MOV' V/4 

such as those listed above. 
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Use and Integration of 
Commercial Off-the-Shelf Software 

Findings (Cont.) 

• DoD has not fully identified the pros and cons associated with the use of COTS 

- Pros 

> Savit monty in d«ign and davalopmcnt of Uioic componitnis 
■ Can Significantly Raducc Dtvalapmcnl Riik 

• Support It CoiKtm 

N Selection of COI'S producli carty in proved cycl» will enable requiretnenCs to be driven by commercial loftware 
capabilltief 

» Very uaefut in rapid prototypirtg 

- 

• Commercial producU nnuil «till be integrated into eyelem and qualified with eyatem 
» Difficulty in Configurement Management and Support for Older Reltaeet 
»• Additional teating may be required to qualify eyetem with commercial compor\enta 
X Subeequent raleaaee determined by vendor, not DoD 
» Security Aspect of Use of COTS Not Well Understood 

• OoD not addressing multiple aapecta 

- D«v«l<ipm«n( anvlroiunanl 
• YadksIcempHltrprsgram 
Vlrua pnUMllon 
C*mm«r<UI Eaplianapt 

• Oaaslfitd software systems s problem f«r commercial compsnles 

• In addition, DoD has not determined when to use COTS 

- Commercial Off-the-Sheif Software ProducU May Not Apply to All OoD Systems 

- Should be used as is • avoid tailoring or special features 

- Most weapon fiyslem/reahUme application software for DoD will not exclusively be "custom," but 
will involve some COTS 


In general, DoD has not identified the pros and cons associated with the use of COTS. DoD must 
learn how to balance the cost savings associated with design and development of commercial software 
products and the significantly reduced development risk with the concern for longer term support and 
system security. Commercial products must still be integrated into and qualified within each defense 
system and there is difficulty in configuration management and support for older releases. The latter point 
is important since many DoD software systems are not deployed before a commercial software product is 
retired or replaced. DoD is not addressing many aspects of the use of COTS software: 

- Development environments 

- Vims protection 

- Commercial espionage 

In essence, DoD has not developed a corporate way to decide when to use COTS software. 
Commercial off-the-shelf software products do not apply to ail DoD systems. Most weapon system/real¬ 
time application software for DoD will not exclusively be "off-the-shelf.” Integration and configuration 
control for COTS software then become important concerns. Further, DoD must learn to use COTS so 
that it avoids tailoring or special featuies. 
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Use and Integration of 
Commercial Off-theShelf Software 


Recommendations 

o Require Trade Studies and Analysis of the Use of COTS in DoD's Software Acquisition 
Process Where Effective 

- Performed by Acquiring Organisation ai Eitonltal Pari of Defining KequiremenU and in Ra^'id Prototyping 
Situationi 

• (imploy Broad Agancy Aiinouncctntnls ur Similar Contractual Approachn to Facilitate Such Sludict 

- Uic of COTS Appropriate When: 

• Delining RequiremtnU 

» Rapid Prototyping Siluatioru 

- Not Required to Tailor C01S Source Code to Application 
> Not Required to be r.rror*Fre« 

• COTS Sollwaie ia *Qom Enough* to Tailor KequiremenU 

• Establish "Customer Friendly" Application-Specific Information Technology 
"Component Stores" 

- Generic AixhiUriurtt for Specific Domaini 

- Rapid RequiremenU Definition Proce' an Prototyping 
ReuaabU^ Prequalified ComponenU 

> Ataemblc Syalems Rather than Develop Them 

- Reduce Lead Time 

- Security ii not Paramount 

• Increase tech base funding for security audit tools for systems employing COTS 

• Capitalize on Innovative Cost-Effective Techniques for Acquiring and Using COTS 
Software Products 

- Sucli as Use of Entetprisa licenses 


Given these findings, the Task Force makes the above recommendations with regard to DoD use 
and integration of commercial off-the-shelf software. The Task Force sees great benefit to be gained 
through exploitation of COTS software; however, DoD must develop corporate approaches to the use and 
integration of COTS software, if it is to gain this benefit. 
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3.5 ACQUISITION 



Acquisition 



• Finding s 

- Acquisition practices have led to: 

» Distance between user and developer 
» Limited participation by commercial software companies 

- Adherence to DoD regulations for reviews and documentation is increasing 
software costs 

» DoD software costs are estimated to be increased by at least 20 % over 
commercial best practice 

» Commercial best practice requires much less documentation than DoD 

- Funding for maintenance planning/execution starts late 
» Maintenance assumed organic; inhibits teaming/partnerships 

- Acquisition focus is on mandatory "how to" specifications and standards 
rather than the product (what) 

» l engthens process and adds costs 
» Discourages harmonization with commercial practice 
» Creates adversarial relationship 

Acquisition process does not reward development of reusable software 




The Task Force was concerned that the specific acquisition and contracting approaches used by 
DoD inhibit use of conimercial practices and software. The Task Force findings in this area ai'e shown 
above. Strict adherence to DoD regulations for reviews and documentation is increasing software costs. 
DoD software costs are estimated to be increased by at least 20% over commercial best practices. The 
DoD focus on detailed technical specifications has lengthened the process, added costs, discouraged 
harmonization with commercial practice, and created a highly adversarial relationship between the 
Government and industry. There is a strong belief that certain commercial companies or divisions of 
companies have opted to stay away from government contracts due to the complexity of the acquisition 
rules and regulations. 
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Recommendations-Acquisition 


• Recommendations 




- Implement the following acquisition approach: 

» Establish acquisition focus on fuixctionality and consistency with 
"commercial best practice" 

» Revise procedures encouraging interaction between user and developer 
and achieving early functionality 

» Minimize DoD regulations for review and documentation that are 
different than "commercial best practice" 

» Require planning for maintenance at beginning of development process 

» Provide government funded vehicle in contracts to incentivize 
development of reusable SW 

- Review all existing military standards and military specifications 
pertaining to software development and documentation, for 
continued applicability, such as DoD-STD 2167 



To this end, the Task Force makes the above recommendations with regard to DoD software 
acquisition practices. In essence, these recommendations can move DoD toward an acquisition approach 
that is more consistent with commercial approaches, while not requiring changes in statutes. 
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To implement this recommendation, the Task Force proposed that DoD evolve a more appropriate 
software life cycle approach as depicted above. This approach provides for early capability (even in the 
initial bidding process) and for a gradual enhancement in capability over time. 
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Acquisition Approach 


Developing the RFP 




Integrated Top Level Specification 

• Ferfonnance 

• Enviroiunent 

• TestA'alidation 


Selection 

Acquiring Agency 

~ Ev«lu«tc Technical Solution and Managemciil Plan 
Evaluate SW Ptoceas Maturity and Pait Performance 
Users 

— Define Requirements 
Evaluate Demnnatrations 
Heavy Participation In Selection 


The Task Force proposed approach to competition is shown in the above figure. The core 
government role in such an approach would be to: 

• Develop the RFP (no “How I'o” statement of work; no management CDRLs; no government in- 
process approvals) 

• Provide an integrated “Top Level” specification (architecture, COTS/reuse, software engineering 
environment and test/validation approach). 

• Employ software meti’ics as a key determinant, 

• Evaluate proposed technical solutions and the proposed management plan 
The contractor would then: 

• Provide an execution plan, management controls and progress miiestones/metncs 

• Describe an in-place, mature software development organization and relevant domain experience 

• Provide a skills matrix describing personnel to be employed 

• Identify a robust development environment and describe applicable prior experience 

• Describe automated process control software 

» Describe the extent to which peer inspections will be used 

• Provide a metrics usage plan and purposes for which they will be used 

• Provide specific reuse and program management plans 

• Propose specific architecture(s) in executable code 
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3.6 SOFTWARE ARCHITECTURE 


Software System Architecture: 
The Missing Link? 


What is software archilecture? 

• Software architecture consists of: 

- Software system components 

- The relationships among those components 

- Rules for their composition (constraints) 

• Defining docuinent would address: 

- System functionality 

- Software system components 

- Interfaces, standards, protocols 

- Execution model 

- Data flows 

- Control flow 

- Critical timing/throughput aspects 

- Error handling 







Software architecture was emphasized by the Task Force as a means for achieving important ends. 
Software architecture consists of software system components, the relationships among these components 
and the rules for their composition (constraints). To use architecture as a management tool, DoD needs to 
define: system functionality along with software system components to be employed; interfaces, 
standards, and protocols to be employed; and the execution model to include data flows, control flow, 
critical timing/throughput aspects, and error handling approach. 















Software System Architecture (Cont.) 


Findings 

• Why is it important? 

- Essential for effective management over the lifecycle 

» Software system lifecycle costs ~ 65% for maintenance 
» ~ 65% of maintenance costs are due to changes/modifications/upgrades 

- Software architecture is a prime enabler of flexibility and reuse 

- Well-formulated architecture might reduce costs of changes/upgrades by 30- 
50% ($4-$7B/year assuming software expense - $30B/year) 


• Why doesn't it play a larger role? 

- Focus is usually on initial cost, schedule, functionality - not lifecycle 

- 2167A reinforces this approach - requires proof that design satisfies 
functionality 

- Difficult to specify, test, etc. 

- Not well understood 

V .. 



Software arcliiteccure is a prime enabler of flexibility and reuse, and a well-formulated architecture 
might reduce costs of changes/upgrades by 30-50% ($4-$7B/year assuming software expense 
~$30B/year). Software architecture has not been emphasized because PM focus has usually been on initial 
cost, schedule, and functionality and not on the life cycle. DoD-STD-2167A reinforces this approach by 
requiring only proof that a design satisfies the required functionality. 









Software System Architecture (Cont) 


Findings (Cont.) 

• Little emphasis on architecture in DoD software programs or regulations 

• Impact of software architecture issue 

- DoD not benefitting from architecture as a key tool for: 

u Evolutionaiy development 

» Early land often) involvement of users with functional capability 
» Ability to include changing commercial technology 
» Reuse 

» Facilitating requirements change with aiinimtun tost and schedule 
» Facilitating product line management 

• Insufficient Progress in: 

- Developing models/standards for domain specific software architectures 

- Open system architectures work 

L___ J 


There is cunently little emphasis on architecture in DoD directives or regulations. As a result, DoD 
is not benefiting from architecture as a management tool. Further, the Task Force sees insufficient 
progress in developing models and standards for domain specific software architectures. 













Software System Architecture (Cont) 



Recommendations 

o Emphasize Use of Software Architecture 

- Establish model and context for architecture selection 

» Standards-based with emphasis on "implementable" 

» Require vendors to propose, manage and control the architecture 

- Require delivery of software architecture definition as first step in any 
software acquisition 

- Foster migration strategies at architecture level 

• Assign responsibility within Government for domain analysis and product 
line developments 

• Provide expertise and resources to ensure coordinated DoD participation 
in commercial/intemational standards bodies and users groups 




The Task Force makes the above recommendations in order to facilitate greater use of software 
architecture as a management tool for DoD software programs and activities. In particular, the Task Force 
sees great value in requiring the delivery of software architecture definition as a first step in any software 
acquisition. Where possible, such software architecture definition should be operational (i.e., executable). 
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3.7 SOFTWARE TECHNOLOGY BASE 


Software Technology Base 


Findings 

• The current DoD software technology base does not 
adequately take advantage of commercial R&D 

• Software technology transfer (internal and external) is not 
receiving adequate emphasis within DoD 

• There is a paucity of data to support prediction of cost 
schedule and performance 


V_ y 


The software technology base (both defense and commercial) provides DoD with ample 
opportunity for significantly improving tlie defense software acquisition life-cycle. 
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Software Technology Base (Cont,) 



Recommendations 







Provide for the evolution of the DoD Software Technology Strategy to alig \ 
with emerging commercial technology and practices 
Emphasize Technology Transfer (External and Internal) 

- Fund technology transfer programs in such topics as: 

» Architectural principles, 

» Architecture description languages, 

» Standard interfaces, and 
» Integration technologies 

- Initiate demonstration programs (e.g., ATDs) to facilitate software technology 
insertion into systems. Examples of candidate crileria; 

» Open standards 
» Use of COTS and COTS 
» Frequent releases to include a number of users 
» Multiple platforms 

» Satisfies commercial standards and interoperability standards across DoD 
organizations 

Initiate fonnal data collection and analysis 


a 



The Task Force makes the above recommendations with regard to DoD software technology base 
investments. The Task Force supports a DoD technology base program that is more closely aligned with 
the wide range of similar efforts ongoing in commercial organizations. 
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4.0 


SUMMARY 


4.1 “ATTA PERSONS” 


"Atta Persons'' 


• Electronic Systems Command Software by Assembly of Modules 

- Software "Component Store" 

• A-109 Acquisition Methodology 

• Dse of Commercial Practices and Products in Reserve 
Component Automation System (RCAS ) 

• Software Reuse, Prototypes and User Involvement of Global 
Command and Control System (GCCS) Initiative 

• PeiTy-Paige Initiaave Toward Migration Systems 


V_y 

The Task Force identified several ongoing efforts worthy of note. 

• The Air Force Electronic Systems Command (ESC) has defined an approach for software system 
development through the assembly of pre-qualified software modules (PRISM), ESC is pursuing 
the development of such software modules. The Task Force was veiy supportive of this program. 

• Office of Management and Budget (0MB) Circular A-109 outlines an approach for major Federal 
system acquisitions that encourages: definition of top level needs vs. detailed specifications; 
exploitation of innovative private sector contributions and use of early competitive demonstrations 
of competing approaches. These all arc acquisition attributes recommended by the Task Force. 

• The Reserve Component Automation System (RCAS) is an ongoing MIS development to support 
the reserves. The program has successfully employed the A-109 acquisition approach and 
extensively used commercial acquisition practices and products. The Task Force commends this 
approach. 

• The Global Command and Control System (GCCS) is an initiative of the Joint Staff (J3 and J6) to 
provide vertical and horizontal interoperability of combat information systems across Services, 
Combatant Commands and Agencies. It was highlighted for its software reuse approach, its use of 
prototypes and its emphasis on operational user involvement. 

• The Perry-Paige migration systems initiative has established a focus on selecting a set of target 
computing systems (including MIS, C3I and embedded) towards which DoD will aim. This 
migration strategy will enable a more cost-effective DoD investment in software across the life- 
cycle. 
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4.2 OVERARCHING RECOMMENDATION 



Overarching Recommendation 



• SECDEF Assign USD (A&T) the Responsibility for DoD-Wide Software 
Technology/ Policy, Practices and Acquisition 

• This Responsibility Includes Allocating Resources and Assigning 
Responsibility for: 

- Drafting and Institutionalizing a New Software Acquisition Policy That 
Includes the Recommendations of this DSB Task Force 

- Creating Incentives and Ensuring Compliance to the Policy 

- Creating Terms and Conditions and Financial Rewards to Maximize the 
Ability of DoD to Realize the Benefits of Commercial Best Practices 

- Requiring Source Selection Evaluation of Development Contractors Through 
a Formal Software Process Capability Evaluation 

- Advising the Acquisition Executive on Matters Conccrrting Software 
Technology, Acquisition and Architecture for Major Programs 

- Maintaining a Digest of Lessons Learned and Best Practices and 
Communicating the Same to Program Managers and Contractors 


Throughout its deliberations, the Task Force acknowledged that the issues associated with defense 
software were applicable across the spectrum of DoD software intensive systems. The Task Force also 
frequently learned of obstacles based, in part, on the current dichotomous DoD structure associated with 
software technology, policy, practices and acquisition. In order to ensure that the DoD reap maximum 
benefit from its recommendations, the Task Force formulated the above overarching recommendation. 
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Overarching Recommendation (Cont,) 


• In Carrying Out This Responsibility, Consider Forming an Executive 
Council 

- Including the DDR&H, the ASD (C31) and Appropriate Representatives from 
the Services and Defense Agencies 

- Provide Supporting "Process Action Team" to Assist in Implementation 


The Task Force also identified the mechanism by which its overarching recommendation could be 
readily implemented. 



APPENDIX A 
TERMS OF REFERENCE 



THE UNDER SECRETARY OF DEFENSE 



ACQUISITION 


WASHINGTON. DC 20301-3000 


M. 0 6' 1993 


MEMORANDUM FOR CHAIRMMI, DEFENSE SCIENCE BOARD 

SUBJECT: Terms of Reference—Defense Science Board Task Force on 

Acquiring Defense Software Commercially 


You are requested to form a Defense Science Board Task Force 
on Acquiring Defense Software Commercially. Determine the 
conditions under which the procurement of defense software (i.e., 
operational software, support software, and software tools) can 
appropriately use commercial practices and develop a strategy for 
defense software procurement that substantially incorporates such 
practices. Specifically address within this strategy DoD use of 
commercial software products. 

The scope of this effort should include all DoD systems that 
are software intensive. It should address all stages in the life 
cycle of a software component from initial procurement to 
evolutionary upgrade of software or of software/hardware 
combinations. This commercially-based strategy should not be 
constrained by existing DoD standards? it should be viewed as a 
coexisting alternative to, rather than replacement of, the 
current DoD procurement strategy. Accordingly, you should 
compare your proposed strategy to the current DoD strategy to 
indicate circumstances in which each strategy is most beneficial. 

To assess the appropriateness of DoD use of commercial 
practices, the Task Force should identify and apply objective 
measures such as elapsed development time, software life-cycle 
cost, management risk, as well as measures of software product 
quality. 

The Task Force should consider at least the following 
topics: 


• Technical: state-of-the-art and best commercial 
practices, development tools, reusable software components, 
and techniques and tools for tailoring commercial software 
components for use in defense systems. 

• Management: DoD management of the commercial development 
process, software risk management techniques and supporting 
tools, minimum delivery time, affordability, maintenance 
after product delivery, post-deployment product enhancement, 
software process support tools, quality and assured 
availability, and use of development and maintenance tools. 









• Contracting: technical data rights, intellectual 
property rights, liability, alternative forms of procurement 
agreements, and incentives for creation of reusable software 
components as well as their subsequent reuse. 

The Director, Defense Research and Engineering will sponsor 
this study. Dr. George H. Heilmeier and Dr. Larry E. Druffel 
will serve as Co-Chairmen. The Office of the Director, Defense 
Research and Engineering will provide the necessary funding and 
support contractor arrangements. The Executive Secretary will be 
Ms. Virginia L. Castor. Commander Robert C. Hardee will be the 
Defense Science Board Secretariat representative. It is not 
anticipated that this Task Force will need to go into any 
"particular matters" within the meaning of Section 208 of Title 
18, U.S. Code, nor will it cause any member to be placed in the 
position of acting as a procurement official. 















APPENDIX B 
BRIEFINGS 

PROVIDED TO THE TASK FORCE 








Briefings Provided to the Task Force 


Joint Staff Views on How DoD Can Adopt Commercial Software 
Practices 

MG David Kelley, Vice Director J6 

Army Views on How DoD Can Adopt Commercial Software Practice 

LTG Peter Kind, Director, Information Systems for 

Naval Views on How DoD Can Adopt Commercial Software Practice 

RADM John Hekman, Commander, NISMC 

Air Force Views on How DoD Can Adopt Commerciai Software 

Practices 

Mr. Lloyd Mosemann, Deputy Assistant Secretary of Air 

Force (Communication, Computers & Support Systems) 

DISA Views on How DoD Can Adopt Commercial Software Practices 

Dr, Mark Schcr, Director of Infrastructures. Defense 

Information Sv.stems Agency 

DoD 5000 Series Resulations 

Mr. Gene Porter, Director, Acquisition Program Integration 

DoD 8000 Series Regulations 

Mr. Harry Pontius, Director, C^I Policy 

DoD-STD-2167A/MIL-STD-498 

Dr. Singh, Senior Manager, Space & Naval Warfare Systems 
Command 

Draft White Paper on Software Acquisition 

Dr. Jack Ferguson, Program Manager, Software Engineering 
Institute 

Data Modeling vs. Data Standards 

LTG Peter Kind, Director, Information Systems for 

Case Study: B2 

Mr. Fred Schwartz. Director of Engineering. 

Case Study: Ensemble of Real-time Software Systems 

MajGen Israel, Director, Defense Airborne Reconnaissance 
Office 

Case Study; Reserve Component Automation System (RCAS) 

MG Gary Stemley, Program Manager for RCAS 

Case Study; AEGIS 

CAPT. Richard Cassidy. AEGIS Technical Director 

Case Study; PRISM 

Mr, Robert Kent, ESC/ENS 

Case Study; BMC3 

Lt Col Robert Phelps. BMDO 

Boeing Views on DoD vs. Commercial Software Practices 

Mr. John Hanson, Director. Systems Software & Engineering. 
Boeing 

McDonnell Douglas Views on DoD vs. Commercial Software Practices 

Mr. L. George Hite, Senior Manager, McDonnell Douglas 

IBM Views on DoD vs, Commercial Software Practices 

Mr. Dave Coyer, Program Manager, IBM Federal Systems 
Company 

Object Technology and a COTS Product Called "SNAP" 

Mr, Joe Fox, Chairman, Template Software Inc. 

Procurement Regulations/Issues 

Mrs. Eleanor Spector, Director, Defense Procurement 
OUSD(A&T) 

Integrated Computer Aided Software Engineering (ICASE) Program 

Col Gary Case, ICASE System Program Director 

Computer Sciences Corporation Case Study on Commercial Software 
Development 

Mr. Daniel Kemp, Senior Parmer. Computer Sciences Corp. 

Commercial Software Acquisition Practices 

Mr. Alex Monow, General Manager, Lotus Development 

Coro. 

Experiences in DoD Software Maintenance 

Mr. Bobby McDonald, Deputy Director, Electronic Warfare 
Directorate, Warner Robins ALC 

MITRE Corporation Experiences in Exploiting Commercial Practices 

Mr. Steven Crisp, Dept. Head, MITRE Corp. 

FAA Study on the Use of Commercial Software 

Mr. John Stenbit, Vice President & General Manager, TRW 
Systems Integration Group 

NSIA Study on the Use of COTS Software 

Gen. William Richardson, USA(Ret), Executive Vice President 
Army Programs, Burdeshaw Associates, Ltd. and Ms. Linda 
Connor, Program Manager, Rockwell International 

DoD Software Reuse Initiative 

Ms. Linda Brown, ODASD(IM)/Information Technology 

MITRE Study on Feasibility of Using a Software Acquisition Maturity 
Model in DoD 

Ms. Judith Clapp, Division As.sitant, The MITRE Coqi. 

Software Management Metrics and Reliability 

Mr. Douglas Putnam and Mr. Lawrence Putnam, Jr., 

Quantitative Software Management, Inc, 

Innovative Techniques in DoD SW Acquisition: F-22 Integrated 
•Product Team Approach 

Colonel Robert Kayuha 

1 Legal Impediments to Acquiring Defense Software Commercially 

Mr. Robert Gorman, OSD General Counsel 


B-l 

















































APPENDIX C 

COMPARISON OF SOFTWARE 
ACQUISITION METHODS 



Comparison of Software Acquisition Methods! 


Requiremen 

ts Definition 

Best Commercial Practice 

Current DoD Practice 

Requirements based on strategic plan and market 
analysis. 

Requirements based on using command Mission 

Need Statement, master plans, and top-level 
certification. 

Requirements based on life-cycle resource 
constraints. 

Requirements based largely on annual budget 
resource constraints. 

Detailed requirements generated by interdisciplinary 
team including users, domain experts, and system 
engineers. 

Detailed requirements generated by buj'er in 
collaboration with user. Team generally includes 
domain experts and acquisition personnel. 

Buyer, user, and vendor are a team. Attitude cf 
partnership, trust and cooperation. Presumption of 
trustworthiness for reputable commercial 
organizations. 

"Us vs. them" mentality about contractors. 

Government thinks in terms of conh'ol, 
accountability, detailed auditing, and double 
checking. Presumption that contractors cannot be 
trusted. 

Functional specification is modified by knowledge 
of availability of existing products. 

Functional an^/or performance specification; little to 
no regard for existing products. 

Vendors involved early in study, analysis and 
prototyping with emphasis on reuse and evolution 
of existing systems. 

May contract for prototypes, but contractor 
involvement in pre-award discussions is 
discouraged. 

Need is based on business case and decisions are 
based on return on investment. 

("time to make") 

Need based on Mission Need Statement; decisions 
based on need, economics, and politics, ("time to 
field") 

Efficient decision processes. 

Decision processes formal and time-consuming. 

Level of documentation is negotiable based on 
individual user needs and complexity of system 
being developed. 

Extensive (often redundant or unnecessary) 
documentation required under 2167A. 

Tailoring of documentation requirements is often 
minimal or discouraged. 

More detailed analysis of cost versus feature. 
Dropping lower valuc/higher ccyst options or 
reducing requirements is practiced. 

Little or no requirements reductions on high cost 
items. 

More requirements trade-off decisions (involving 
complexity and schedule) for reduced time to field. 

Very little flexibility to trade-off requirements creep 
versus complexity and schedule. 

Selected vendors may assist in preparing 
specifications. 

Vendors not involved in preparing specifications. 

Tools used to create system models for use in 
requirements definition; e.g., GUI Building. 

Requirements definition based on Mission Need 
Statement. 

Flexibility allowed in choice of programming 
language. 


Evolutionary and incremental approach favored. 

Requirements defined up front with little flexibility 
for modifications. 

Summary 

Commercial is more flexible and open between users and suppliers, and requirements are based on a 
strategic plan. In the commercial world, there is more willingness to adjust requirements based on 
availability of products and thus to filed a system sooner and evolve it to include more capability at 
significant cost savings. 


* This appendix was initially derived from a White Paper on software acquisition methods prepared by the Software 
Engineering Institute. The resulting content represents the consensus of this Task Force. 
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Vendor Selection 


Best Commercial Practice 

Current DoD Practice 

Solicit multiple (but not all) qualified vendors -a 
selected few. Encourage teaming with a view to 
attaining a long term relationship that covers the 
entire life cycle and fosters trade-offs in cost and 
schedule. 

Solicit all possible vendors. Vendor proposals must 
meet 100% of requirements. Teaming seldom 
encouraged; development and maintenance usually 
separ ate entities. 

Compare vendor history and experience. Maintain 
long-term relationships. 

Can compare previous performance, but normally 
can't have long-term relationships. 

The organization that will be responsible for a 
system over its full life cycle is heavily involved 
from the beginning. 

Maintenance organization not usually involved in 
vendor selection process. 

Use site visits and demonstrations to gain 
knowledge of vendor capabilities. 

Site visit only by capability evaluation team, or 
other expert teams. Visits are very structured. 

Negotiate for best values based on: (1) trade-offs of 
costs and requirements licensing; and (2) 
consideration of both vendor and buyer best 
interests. 

Negotiation based on lowest cost and shortest 
schedules, endangering maturity of finished 
product. 

Overall goals: (1) obtain product at reasonable cost 
as soon as possible; and (2) achieve the business case 
for the system. 

Overall goal: Obtain lowest cost product that 
rigorously meets all requirements, but be fair. 

Relatively few review and approval steps once 
vendor is selected. 

Review and approval process more structured and 
complex once vendor selected. 


Past performance considered, but only as a minor 
factor. 

More flexibility in vendor selection based on metrics 
and overall assessment. 

Selection of vendor forced by use of pre-defined 
metrics for proposal evaluation. 

Modifications made as procurement proceeds in 
order to get best results. 

Change difficult once process begins. 


Summary 

Very different processes with commercial much more flexible, but with no requirement for fairness, or to 
maintain the public trust. Commercial encourages vendors to offer best solution, but solution may not 


meet 100% of the requirements. Teaming and long-term relationships are more easily accommodated by 
industry. _ 
























1 Development Process | 

Best Commercial Practice 

Current DoD Practice 

Vendor often tailors existing systems and uses 
COTS. System designed to fit in a defined product 
or product line architecture. 

Varies with application. Some systems use COTS. 
However, usually a new system that doesn't reuse 
legacy software. Unique systems are built with little 
regard for architecture. 

Buyer may have heavy involvement in design and 
development as part of the team (Integrated Product 
Development team). 

Formal, structured spiral, or waterfall model. Buyer 
oversees, but team approach is not usually 
emphasized. 

Reviews typically informal and stress progress 
against goals. 

Reviews usually very formal and include technical 
design details in addition to progress metrics. 

Buyer actively involved as co-participant in 
management of technical details. 

Micro management of technical details. 

Heavy user involvement. 

Limited user involvement. Heavy buyer 

involvement. 

Vendor embraces one or more industry standards 
which improves interface and integration with COTS 
products. 

Government and industry standards called out. Not 
all government standards enhanced by COTS 
products. 

Buyer requirements may be translated to more 
"general purpose" requirement for potential 
software reuse. 

Tailored system; little, if any, focus on designing in 
reusable code. 

Management reviews and degree of oversight are 
commensurate with size and risk of program. 

Notably more detailed reviews and oversight 
performed. 

Prototyping common, with joint applications 
development teams (user and developer) working to 
clarify requirements and incorporate new 
requirements that do not affect cost or schedule. 

Prototyping seldom used. 

Use of flexible architectures allows insertion and 
plug and play of COTS products. 

Use of MIL standard computers and legacy 
instruction set architectures restricts new 
devekipment. 

Suxamiiry 

Commercial more flexible with likelihood of a team approach and is biases toward reuse and tailoring of 
existing systems. Multi-year acquisitiorss not re-justified each year. Product impsoveirtients are anticipated. 


I 
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Business Practices 


Best Commercial Practice 


Informal contracting, joint ventures, partnerships 
with mutual economic benefit and vested interest in 
success. 


Oversight built on established relationships. 


Current DoD Practice 


Difficult to write contract to motivate contractors to 
reduce cost; expensive to terminate contractors. 


Burdensome cost accounting procedures required; 
extensive oversight, reporting, and documentation 
requirements. 


Government personnel regulations, policies, and 
practices determine qualifications of its managers, 
rotations of assignment, and training. 


Multi'Vear effort. Yearly, unpredictable fundin 


Summary 

Commercial practice more flexible with greater incentives. 


Can hire and fire vendors and managers. 


Multi-year effort and fundin 


Integration Teotine and Delive 


Best Commercial Practice 


Unless system is for a new plant, then there are 
major "cut-over" issues. 


Sometimes difficult to assemble complete system in 
laboratory environment due to co.st. 


Current DoD Practice 


Similai' "cut-over" or transition issues. 


Usually integrate system in laboratory prior to 
operational testing. 

Development testing vs. operational testing via 
statutory mandate. 


Little beta testin 


Structured, specified operational testing conducted 
by separate authority. Acceptance authority rests 
with buyer. 


Emphasis on DoD as oversight and approval 
author! 


Summary 

Integration and functional testing seem appropriate to the need. DoD use of separate test agency adds time 
and complexity. Absence of beta testing increases cost to DoD. 


Beta testing widely used to quickly find errors. 


Ultimate acceptance authority rests with buyer, not a 
separate organization. 


Buyer/user/vendor are a team. 
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Rights in Data 


Best Commercial Practice 

Current DoD Practice 

If custom development, buyer gets all rights, but 
vendor may retain right to subsequent sales. 

Specified by contract. 

Government usually demands all rights for 
government use due to organic support and 
maintenance needs, and competition (via statutory 
mandate). 

If tailored version of standard system, buyer only 
gets rights to tailored parts. 

Same as commercial. May have exceptions for 
proprietary material. 

Summary 

Similar, but commercial is a little more flexible, especially regarding resales. 


Maintenance | 

Best Commercial Practice 

Current DoD Practice 

Organic support shifting to outsourcing or vendor. 

Organic support, with reluctance to be dependent on 
vendor. Use of depot maintenance makes 
interoperability issues more manageable. Also, 
must be responsive to user for critical systems. 

Level of discrepancy reporting required is based on 
user needs. Problem resolution usually delayed 
until next major release of software. Buyer has 
limited power to implement fast resolution of 
problems. 

Bureaucratic and paper-intensive discrepancy 
reporting and change control board often imposed. 
Award fees may be tied to problem workoff, 
resulting in fast resolution of problems. 

COTS solutions take advantage of economics of 
scale since maintenance costs are leveraged across 
multiple buyers. 

Unique software requires custom maintenance all 
borne by the single buyer. 

Summary 

The DoD and industry currently have different requirements, and trends are moving apart. However, DoD 
is currently reevaluating its practices for hardware systems and perhaps should also reevaluate for software. 
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APPENDIX D 

PROPOSED ACTION PLAN FROM 
SERVICE SOFTWARE EXECUTIVE 

OFFICIALS 








OHict, Oir*c(or Informitton 
Systtmt lor Command, Control. 
Commumcations. Ii Computara 


DEPARTMENT OF THE ARMY 
OFFICE OF THE SECRETARY OF THE ARMY 
107 ARMY PENTAGON 
WASHINGTON DC 20310-0107 



MEMORANDUM FOR THE EXECUTIVE SECRETARY. DEFENSE SCIENCE BOARD 
TASK FORCE ON DOD ACQUISITION OF COMMERCIAL SOFTWARE 

SUBJECT; Proposed Action Plan with Regard to Common Changes Proposed by DoD 
Speakers to Revise the Software Acquisition Process 


The DSB Task Force asked us to provide a list of common recommended 
changes needed to revise DoD's software acquisition process. Our recommended list 
of proposed changes is provided in the enclosure to this memorandum. As we worked 
together to develop this list, we found it useful to categorize the recommended changes 
into five major categories. Several of our recommendations did not receive adequate 
emphasis in our presentations, yet are important to the process we are studying. We 
feel that if OSD is serious about addressing Software Acquisition issues, a single 
senior office with primary responsibility for effecting the reforms proposed by the DSB 
will be essential. Further, such an official or office must receive “from the heart" 
sustainment and backing from the highest levels within OSD. The enclosed list 
addresses our concerns vyith regards to the questions before the DSB. We hope that 
this will be beneficial to the DSB Task Force and we all look forward to continued active 
participation in this effort. 



PETER A. KIND 
Lieutenant General, GS 
Director of Information 
Systems foi' Command, Control, 
Communications and CoiTiputers 



Deputy Assistant Secretary 
(Communications, Computer, and Support 
Systems) 


Date 


4 Feb 94 


Date_ 



^oTTiTG. hekman 

Rear Admiral, SC, USN 


Commander, Naval Information 
Systems Center 


Date 


4 l^eb 94 
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JOINT RECOMMENDED PROPOSED ACTIONS 

I. DESIGNATE SINGLE. COMMON DOD SOFTWARE MANAGEMENF OFFICIAL 
(ACTION OFFICE (USD)) 

A. ESTABUSH DEPUTY ASSISTANT SECRETARY OF DEFENSE 
(SOFTWARE) 

B. PROVIDE SUFITCIENT RESOURCES FOR MISSION 

II. SOFTWARE ACQUISITION AND UFE-CYCLE MANAGEMENT (ACTION OFHCE 
NEW DASD) 

A. PROMOTE REUSE BASED ON DOMAIN MANAGEMENT 

1. Define domains of interest or "areas of business" 

2. Assign responsibilities to manage the domains 

3. Establish and manage common architectures within the domains 

E. REDUCE MONITORING OF MATURE COMPANIES 

1. Identify criteria for assessing/detennine process maiurity 

2. Consider maturity in source selection 

3. Allow sole source extension for high quality vendors 

C. PROMOTE GOVERNMENT/INDUSTRY TEAMING 

1. Reduce Documentation Requirements to a minimum 

2. Provide for electronic deli very/evaluation/exchange 

D. MANAGE RISK 

1. Mandate the use of market research 

2. Use metrics effectively for management 

3. Develop Risk Management Disciplines 

4 . Stick with a winner/punish poor perfonners 

III. POLICY AND STANDAJIDS 

A. ADOPT COMMERCIAL STANDARDS (ACTION OFnCE(ASD(C31))) 

1. DoD invests in commercial standards developments 

2. Change FAR/DFAR and SD-1 as appropriate 

B. REVISE THE MILESTONES FOR SOFTWARE-INTENSIVE 
DEVELOPMENTS (ACTION OFHCE USD(A&T)/ASD(C3I)) 

1. Address the need for a software-first philosophy 

2. Provide for a layered software/hardware standards based architecture 











C. DEVELOP DOD "BETA TEST" PHILOSOPHY (ACTION OFFICE OSD(T&;E) 

1. Team with Universities/other appropritue activities 

2. Allow l ieldiiia of software direct from test beds with user consent 

IV. PERSONNEL 

A. DEVELOP SOFTWARE ACQUISITION MANAGERS (ACTION OFHCE 
USD(A<tT)) 

1. Provide a career path for software managers 

2. Integrate software personnel into senior acquisition staff 

13. PROVIDE SOFTWARE EDUCA'HON FOR SENIOR MANAGERS 
(ACTION OFFICE USD(A&T)) 

1 Develop DoD Senior Software Managers Course 
2. Develop and provide interactive training tools for 
senior managers to pel feet software management skills 

C. PUBUSH AND PROMOTE "BEST PRAC'HCES" HANDBOOK (ACllON 
OFFICE OSD{T&E)) 


D. ENSURE DOMAIN EXPERTISE (ACllON OFFICE MIUTARY 
DEPARTMENTS/ AGENCIES) 


V. SOFTWARE TECHNOLOGY BASE AND TRANSITION (ACTION OPHCE 
DDR&E) 


A. PROVIDE FOR SOFTWARE TECHNOLOGY INSERTION INTO SYSTEM 
ACQUISITION 

B. INVEST IN THE DOD SOFTWARE TECHNOLOGY STRATEGY 

. C. PROVIDE FOR THE EVOLUTION OF THE DOD SOFIWARE 
TECHNOLOGY STRATEGY TO CAPTURE EMERGING COMIvxERCIAL 
PRACTICES 





